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Abstract 

A human-in-the-loop simulation experiment was 
designed and conducted to evaluate an airborne 
self-separation concept. The activity supports 
the National Aeronautics and Space 
Administration ’s (NASA) research focus on 
function allocation for separation assurance. 
The objectives of the experiment were twofold: 
(1) use experiment design features in common 
with a companion study of ground-based 
automated separation assurance to promote 
comparability, and (2) assess agility of self- 
separation operations in managing trajectoiy- 
changing events in high traffic density, en-route 
operations with arrival time constraints. This 
paper describes the experiment and presents 
initial results associated with subjective 
workload ratings and group discussion feedback 
obtained from the experiment’s commercial 
transport pilot participants. 

1 Introduction 

In current Air Traffic Control (ATC) operations, 
the traffic separation function resides with air 
traffic controllers, not with controller 
automation or with aircraft. The controller 
visually monitors the position of traffic within a 
sector at all times, makes predictions on when 
and where aircraft might lose separation, 
determines resolutions to these conflicts, and 
issues instructions for rerouting or maneuvering 
to pilots by voice radio. Using primarily a radar 
display and a computer repository of flight plan 
information, the controller provides separation 
in a mostly manual way, with minimal reliance 


on tools designed and intended to support the 
function [l]. 1 

Aircraft today also contribute little to the 
function of traffic separation. Though many 
commercial aircraft are equipped with a Flight 
Management System (FMS) and thus capable of 
high-performance navigation, the flight crew’s 
ability to use this capability to provide 
separation from traffic on their own is 
essentially nonexistent. Without onboard traffic 
surveillance, they cannot detect conflicts or 
make trajectory modifications safely with 
respect to traffic beyond visual range. 
Therefore, today’s air traffic controllers remain 
solely responsible for separation of traffic 
operating under Instrument Flight Rules (IFR). 

Air traffic controllers expertly provide safe 
operations at current traffic levels, but the 
method restricts the ability to indefinitely 
accommodate expected levels of traffic growth 
[2]. The limiting factor is controller workload, 
which governs the number of aircraft and the 
configuration of traffic flows that can be 
monitored and controlled simultaneously within 
a sector [3]. Traffic capacity is limited 
primarily by the human perception of traffic 
flow complexity and the controller’s ability to 
maintain complexity at a manageable level. In 
addition, the ability to respond with agility to 
dynamic conditions is restricted by the 
requirement for serial communications in which 
controllers issue relatively tactical instructions 


1 ATC has some automation systems they use, such as 
Short Term Conflict Alert (STCA) and Minimum Safe 
Altitude Warning (MSAW), that are rudimentary and 
used as a backup to the controller. 


WING, MURDOCH, ET AL. 


to one aircraft at a time. These procedures slow 
down the process of traffic management and 
therefore impede growth in air traffic volume. 

New technologies, such as airborne and 
ground-based separation automation tools and 
airborne surveillance, provide potential means 
for overcoming these restrictions by exploiting a 
reallocation of functions between humans and 
automation and between ATC and aircraft. The 
current separation function is weighted toward a 
human-centered, ground-based allocation. 
Through application of advanced computer 
automation, aircraft trajectories can be modeled 
with much greater precision and monitored 
automatically for traffic conflicts. 2 Resolutions 3 
can be rapidly computed and sent in full form to 
the aircraft’s FMS for execution by the flight 
crew. In a ground-based application of this 
concept currently under study [4] [5], the 
separation function resides in the ATC facility 
but is largely automated, and trajectory 
revisions are sent automatically to aircraft via 
data link. Controllers monitor the operation and 
provide services when needed, typically on an 
exception basis. 

Emerging airborne surveillance technology 
enables a further allocation of the separation 
function from the ground to the aircraft [6]. 
Automatic Dependent Surveillance Broadcast 
(ADS-B) provides the means for an aircraft to 
directly receive position and trajectory 
information from broadcasting aircraft. This 
information, supplemented by uplinked radar 
data of non-ADS-B-equipped aircraft, allows 
receiving aircraft to host the automation 
functions that predict trajectories, detect 
conflicts, and compute resolutions. Thus, the 
aircraft can theoretically provide the separation 
service directly for themselves. The concept is 
referred to as self-separation and is also under 
study [7] [8]. Self-separation does not require 
ground-based automation for separation or the 
infrastructure for uplinking trajectories into the 


2 A conflict is a predicted loss of separation (LOS) with 
another aircraft. LOS is typically defined by a lateral and 
vertical standard, e.g., five nautical miles (nmi) laterally 
and 1000 feet (ft) vertically. 

3 A resolution is an FMS trajectory revision or a tactical 
maneuver that eliminates a conflict. Resolutions may be 

in the lateral, vertical, or speed dimensions. 


FMS. 4 The onboard automation system can be 
integrated directly into the avionics system to 
provide more accurate trajectory prediction and 
more resolution alternatives to the flight crew. 

A human-in-the-loop (HITL) simulation of 
self-separation was recently conducted in 
support of the NASA research focus on function 
allocation for separation assurance. The 
objectives of the experiment were twofold: (1) 
use experiment design features in common with 
a companion study of the ground-based 
automated separation concept to promote 
comparability, and (2) assess agility of self- 
separation operations in managing trajectory- 
changing events in high traffic density, en-route 
operations with arrival time constraints. This 
paper describes the experiment and presents 
initial subjective response data. Section 2 
summarizes the self-separation operational 
concept as presented to the experiment’s 
commercial transport pilot participants. 
Sections 3 through 6 present the experiment 
objectives, scenarios, simulation platform, and 
experiment design. Section 7 presents initial 
results associated with subjective workload 
ratings and group discussion feedback provided 
by 46 subject pilots. 

2 Operational Concept 

In NASA’s self-separation concept, an aircraft 
operating under self-separation is said to be 
operating under ‘Autonomous Flight Rules’ 
(AFR). AFR operations would occur primarily 
in the en-route phase of flight but could also 
include segments of the departure and arrival 
phases. They may occur in homogeneous 
airspace or mixed-operations airspace, i.e., AFR 
and ground-managed aircraft sharing the 
airspace without segregation. 

AFR can be considered by operators as an 
additional flight fding option to Visual Flight 
Rules (VFR) and IFR. AFR operations would 
resemble VFR operations in that responsibility 
for separation from traffic lies with the pilot, not 


4 Ground-based services would still be required by these 
aircraft for other purposes, for example managing their 
insertion into the schedule of converging arrivals to high- 
demand terminal airspace. 


2 


FUNCTION ALLOCATION WITH AIRBORNE SELF- 
SEPARATION EVALUATED IN A PILOTED SIMULATION 


the controller. AFR operations would resemble 
IFR operations in that they may be conducted in 
Class A airspace in any meteorological 
conditions, and that appropriate standards for 
separation would apply unless aircraft are 
visually acquired. An AFR pilot has complete 
authority to manage and revise the aircraft’s 
trajectory and is therefore free to choose a flight 
path and altitude, provided that traffic 
separation requirements are met. The pilot 
works with an onboard AFR tool (i.e., 
automation) to accomplish the task. 5 

2.1 AFR Rules for Pilots 

The operational concept evaluated in this 
experiment included four AFR ‘rules’ presented 
here in priority order. The functionality of the 
AFR tool that supports the pilot in adhering to 
these rules is presented in Section 5.2. 

AFR Rule #1 : The pilot is to resolve traffic 
conflicts without delay when notified. The pilot 
does not scan a traffic display for conflicts, but 
rather relies on the AFR tool for conflict 
detection. When notified of a conflict by the 
AFR tool, the pilot is expected to select from a 
set of tool-provided resolutions and is expected 
to execute the selected resolution in a timely 
fashion. Fig. 1 shows an example of a conflict 
(yellow segment and aircraft) and a resolution 
(blue line). 

AFR Rule #2 : The pilot is to use the AFR 
tool to check trajectory changes for conflicts 
before executing the change. Trajectory 
changes are made for many reasons besides 
conflict resolution, such as turbulence, 
unexpected headwinds, weather hazards, fuel 
efficiency, and absorbing arrival delays. Any 
time the pilot intends to initiate a flight profile 
change, the new route or maneuver must first be 
probed for traffic conflicts. AFR Rule #2 
prohibits a trajectory change that would create a 
‘Level 2’ conflict alert for any aircraft, 
including their own. This notification level 
indicates greater urgency to resolve, because of 


? This automation tool is specific to the separation 
function. Separate tools, such as the Traffic Alerting and 
Collision Avoidance System (TCAS), remain focused on 
their designed function of providing collision escape 
maneuvers. 



Fig. 1 Navigation display showing a 
conflict (yellow) and a resolution (blue). 

reduced time to LOS. Less urgent ‘Level 1’ 
conflicts may be temporarily created but must 
then be resolved according to AFR Rule #1 . 

AFR Rule #3 : The pilot is to ensure that 
the aircraft’s trajectory will conform to ATC 
constraints, should any be in effect. ATC 
constraints are sometimes necessary to protect 
special use airspace or to manage the traffic 
flow rate into capacity-constrained areas such as 
terminal airspace. In this latter case, ATC may 
assign crossing times at an arrival metering fix 
to inbound aircraft, the pilots of which then 
enter into the FMS their ‘required time of 
arrival’ (RTA) constraint at that fix. Any 
changes to the trajectory must ultimately 
support RTA conformance, otherwise ATC 
must be notified if the constraint becomes 
unachievable. 

AFR Rule #4 : The pilot is to have the auto- 
flight system remain FMS-coupled as much as 
possible. This lowest priority rule, perhaps 
better described as a guideline, is intended to 
maximize the proportion of aircraft on a 4D 
trajectory, i.e., ‘strategic’ flight, while providing 
the pilot with the flexibility to operate in 
‘tactical’ flight when necessary, i.e., using just 
Mode Control Panel (MCP) guidance. Aircraft 
flown using FMS guidance are anticipated to 
have more predictable and stable trajectories, 
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thereby benefitting all operators and controllers 
using that airspace. AFR pilots are therefore 
encouraged to fly with the FMS engaged to the 
greatest extent reasonably achievable. 
Specifically in conflict situations, aircraft in 
strategic flight are given higher priority over 
aircraft in tactical flight. 

2.2 Coordination and System Optimization 

Although the concept of self-separation may 
seem very vehicle-centric, inter-aircraft 
coordination and system optimization also play 
central roles. Coordination is applied implicitly 
through, for example, right-of-way rules. In 
every conflict, a common rule set defines one 
aircraft as having priority and the other as the 
‘give way’ vessel (following in the original 
nautical tradition). In AFR, this priority is 
manifested in the timing of notifications such 
that the ‘give way’ aircraft crew is notified first 
and generally resolves the conflict by itself. If 
time passes without resolution, the priority 
aircraft crew is then notified, providing a 
valuable system redundancy and safety benefit. 
Right-of-way rules would generally be based on 
conflict geometry, but may also be designed to 
incorporate a variety of additional system-level 
optimization objectives, such as giving arrival 
traffic flows priority over crossing traffic. Since 
the rules are encoded directly in the automation 
and do not rely on human memory or 
interpretation, the rule set may be as complex 
and extensive as needed to provide the desired 
system-level balance of equity and efficiency. 

Additional means for system optimization 
are also present, although in a manner different 
than in centralized systems. For instance, 
overall operations cost is minimized by the 
flight crew directly through their trajectory 
authority, as this is nearly always one of their 
principal objectives. Contention for airspace 
and other resources is minimized through the 
application of ATC constraints such as RTAs. 
System predictability is enhanced by AFR 
aircraft receiving elevated priority when using 
FMS guidance. Although total system behavior 
of AFR operations is still under study, these 
elements of the concept are expected to be 
beneficial at the system level. 


3 Experiment Objectives 

This AFR experiment was designed to meet two 
objectives. The first objective was to determine 
the degree of comparability that could be 
achieved between experiments in separate HITL 
laboratories that were conducted to study 
different approaches to separation assurance 
under 'four-dimensional trajectory -based 

operations' (4D TBO). This objective was 
derived from a NASA programmatic goal to 
investigate issues of air/ground function 
allocation between the airborne and ground- 
based 4D TBO concepts developed and matured 
by NASA to a medium-to-high level of fidelity 
by different research teams using different 
laboratory facilities. Two experiments were 
therefore designed, one in each laboratory, with 
subject pilots in the airborne self-separation 
experiment and subject controllers in the 
ground-based automated separation experiment. 
The experiment designers took the initial steps 
toward comparability by using common 
experiment design matrices, identical initial 
traffic scenarios, and jointly defined metrics. 
While these steps do not guarantee truly 
comparable results, the exercise was expected to 
shed light on issues of comparing significantly 
different operational concepts in dissimilar 
simulation platforms with different human 
subject populations (i.e., commercial transport 
pilots vs. air traffic controllers). The resulting 
comparability assessment is not the subject of 
this paper but was a significant factor in the 
experiment design presented here. 

The second objective of the AFR 
experiment was to assess agility of self- 
separation operations under high traffic density 
conditions. As traffic levels rise, it remains 
important that a trajectory management system 
be able to react nimbly to events requiring many 
aircraft to change trajectories nearly 
simultaneously. In the self-separation concept, 
the trajectory management system is distributed 
among the aircraft, and it is expected that the 
timing of trajectory-change events would not 
have an impact on performance, whether these 
events be dispersed in time or synchronous. 6 


' The same might not be true for a controller, who must 
issue trajectory change instructions in a serial fashion. 
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The experiment sought to create an 
operationally realistic event (to both pilots and 
controllers) requiring a trajectory change to be 
made by a large number of aircraft and to 
control the timing of that event. The event 
chosen was notification of an arrival delay 
necessitating a maneuver to lengthen or stretch 
an aircraft’s flight path to absorb the delay. 
RTA change notifications were sent as data link 
messages. Timing was controlled by scheduling 
the messages to be synchronous among aircraft 
in some scenarios and dispersed in others. 

4 Scenario Description 

The airspace modeled was a rectangular region 
defined by the corner points 36. 5N 94W and 
40. 5N 86W. This area, approximately 376 nmi 
by 240 nmi, includes the eastern region of 
Kansas City Center and the western region of 
Indianapolis Center, and it is roughly centered 
over St. Louis, Missouri. The size 
accommodated 30-minute flights for the subject 
pilots (and controllers), and the location 
provided a mix of crossing and transitioning 
traffic. Vertically, the experiment airspace 
extended from Flight Level (FL) 290 to FL 400, 
although climbing and descending aircraft were 
permitted below the floor. City pair routing was 
roughly similar to today, including the use of 
existing waypoints and standard terminal arrival 
routes. Although all aircraft were initiated at 
regular 1000 ft altitudes (FL340, FL350, etc.), 
aircraft were permitted under AFR rules to 
change to any altitude during the scenario (e.g., 
FL362). Similarly, changes to routing were 
permitted and expected. The simulated winds 
were westerly from 30 to 50 knots with small 
variations in magnitude and direction by altitude 
but no variation laterally or with time. Wind 
forecasting errors were not modeled. 

In every scenario run, each subject pilot’s 
aircraft was initialized in a level cruise 
condition near the FMS-computed optimum 
altitude and approximately 25-30 minutes from 
the top-of-descent point, i.e., close enough to 
the destination to realistically expect an RTA 
assignment for arrival metering. Initial RTA 
assignments were set to be approximately equal 
to the aircraft’s estimated time of arrival, with 


adjustments made only as necessary to avoid 
bunching or overscheduling the arrival flow (for 
subject controller realism in the companion 
ground-based experiment). Initial RTAs and 
RTA changes, when they were used, were sent 
to selected aircraft at the appropriate time via a 
data link. 

5 Experiment Environment 

The self-separation experiment was conducted 
in the Air Traffic Operations Lab (ATOL) at the 
NASA Langley Research Center, in Hampton, 
Virginia. This facility specializes in simulation- 
based research and development of advanced 
operational concepts employing ADS-B, but is 
suitable for a variety of operations research. For 
conducting batch and HITL experiments, it has 
over 300 computers, including 12 desktop pilot 
workstations, four of which are shown in Fig. 2. 



Fig. 2 Desktop pilot workstations in the 
NASA Langley Air Traffic Operations Lab. 


5.1 Simulation Platform 

The ATOL computing network hosts the 
Airspace and Traffic Operations Simulation 
(ATOS) simulation platform [9]. ATOS 
provides a medium fidelity setting for studying 
the interactions of aircraft in a realistic ADS-B 
environment. ATOS uses an implementation of 
High Level Architecture (HLA) to network 
together multiple individual Aircraft 
Simulations for Traffic Operations Research 
(ASTORs) and a background traffic generator, 
Traffic Manager Executable (TMX) [10], 
ATOS supports the exchange of industry 
standard ADS-B reports (i.e., state vector, mode 
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status, air referenced velocity, target state, and 
trajectory change reports) with standard 
message content and broadcast frequency [11]. 
For this experiment, frequency interference 
modeling was disabled, transmission range was 
fixed at 120 nmi, broadcast rate of key reports 
was established at 1 Hz, and trajectory intent 
was broadcast up to 12 trajectory change points 
(i.e., effectively full trajectory intent). 

The traffic aircraft in the experiment 
scenarios were provided by 12 HITL ASTOR 
computers, 63 batch ASTOR computers, and up 
to 557 lower fidelity aircraft modeled by a 
single TMX computer. Since the experiment 
was designed to model homogeneous 
operations, all aircraft operated under AFR. 
The function of the hundreds of TMX aircraft 
was to create the required traffic density and to 
provide ADS-B message broadcasts of aircraft 
performing AFR procedures. For this purpose, 
TMX aircraft included capabilities for conflict 
detection and resolution; however, their 
separation performance was not the subject of 
this experiment. 

Each ASTOR computer simulates one 
aircraft at medium fidelity. ASTOR has 
realistic displays and controls representative of 
a Boeing 777 and a six degree-of-freedom flight 
performance model representative of a medium- 
sized twin-engine transport aircraft. For the 
functions required in this experiment, the auto- 
flight system and FMS were fully functional. 
Its avionics bus emulates ARINC 429 
specifications for realistic internal and external 
communications [12]. ASTOR supports 
trajectory uplink and auto-load, a capability in 
this experiment used for loading RTAs. In 
HITL mode, ASTOR is controlled by subject 
pilots using a desktop computer mouse. In 
batch mode, a pilot model automatically 
operates ASTOR controls according to standard 
AFR procedures. 

5.2 AFR Tool 

ASTOR contains a research-prototype AFR tool 
for the pilot called the Autonomous Operations 
Planner (AOP) [13]. AOP supplied the self- 
separation automation functions necessary for 
the pilot to meet the aforementioned four AFR 


rules: (1) resolve conflicts when notified, (2) 
check and clear all trajectory changes for 
conflicts before executing, (3) conform to ATC 
constraints, and (4) remain FMS-coupled 
whenever possible. The method of AOP 
support for each rule follows. 

Supporting AFR Rule #1, AOP automated 
the process of conflict detection and crew 
notification by making trajectory predictions of 
the ownship and all traffic aircraft within ADS- 
B reception range, and by probing these 
trajectories for predicted loss of separation 
(LOS). A nominal look-ahead horizon of 10 
minutes was used for detection, and the 
trajectory predictions were refreshed at least 
every 10 seconds. Volumetric uncertainty 
bounds were applied to all trajectory segments 
to encompass prediction errors and minimize 
missed alerts [14]. AOP notified the pilot of 
conflicts using textual, aural, and graphical 
methods, and at a time and notification level 
appropriate to the ownship aircraft’s right-of- 
way and the conflict’s level of urgency. Fig. 3 
shows the staggered notification scheme used to 
provide implicit coordination and minimize 
simultaneous resolution actions by both aircraft. 


Time to Loss 
of Separation 
(minutes) 

Right-of-Way 

Give Way 
Aircraft 

Priority 

Aircraft 

10 to 7 

Level 1 

“Traffic Conflict” 

No alert 

7 to 6 

Level 1 

“Traffic Conflict” 

6 to 5 

Level 2 
“Traffic Alert” 

5 to 0 

Level 2 
“Traffic Alert” 


Fig. 3 Conflict notification based on right-of- 
way to reduce simultaneous resolutions. 


Also for AFR Rule #1, AOP automated the 
process of computing acceptable resolutions to a 
conflict. Two methods were available to the 
pilot: strategic [15] and tactical [16]. A 
strategic resolution is a modification of the FMS 
route, and in most situations, AOP will present 
both lateral and vertical alternatives. 7 For each 
of these alternatives, a pattern-based genetic 
algorithm determined the fuel-optimal solution, 


7 Speed resolutions were not employed in this experiment. 
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and the pilot could upload either option to the 
FMS for execution. Fig. 1 shows a Level 2 
“traffic alert” and an AOP-computed lateral 
strategic resolution that is ready for upload and 
execution in the FMS. To execute a vertical 
resolution, the pilot must also reset the MCP 
altitude window appropriately for the new 
altitude. A tactical resolution is a non-FMS 
maneuver activated by the pilot typically 
through MCP ‘track select’ or ‘flight level 
change’ commands. Again, AOP will typically 
offer both alternatives, computed by sweeping 
through possible track, altitude, and vertical 
speed changes until a conflict-free maneuver is 
found. At five or four minutes to LOS for the 
give-way and priority aircraft, respectively, 
AOP will enter ‘tactical override’ mode and 
offer only tactical resolutions, given the short 
time remaining to resolve the conflict. 

To help pilots meet AFR Rule #2 (i.e., 
clear all trajectory changes before executing), 
AOP probed ‘provisional’ (i.e., “what if...”) 
trajectories or maneuvers prior to execution. 
‘Planning’ conflict symbology was displayed 
that indicated whether the proposed change 
would create a conflict and at what notification 
level. With this capability, pilots could safely 
investigate both FMS strategic trajectory 
changes and MCP tactical maneuvers before 
acting. In addition, AOP displayed yellow 
bands on the flight displays to indicate ranges of 
tracks and vertical speeds that should not be 
selected since their selection would result in a 
Level 2 conflict for themselves or another 
aircraft. Fig. 1 shows such a ‘maneuver 
restriction band’ on the compass rose of the 
navigation display. 

AFR Rule #3 (i.e., conform to ATC 
constraints) was supported with a combination 
of FMS and AOP capabilities. The FMS 
evaluated any required waypoint constraints, 
such as an RTA, and notified the pilot if the 
aircraft could not meet a given constraint with 
speed changes alone, e.g. too much delay to 
absorb by simply slowing down. AOP then 
provided the capability to ‘resolve’ this ‘unable 
RTA’ situation with path changes using the 
same strategic resolution capability for 
resolving traffic conflicts. In this experiment, 
only lateral ‘Resolve RTA’ solutions were 


enabled. AOP will also resolve conflicts with 
airspace regions defined by polygons; however 
no such regions were included in this 
experiment. 

AOP supported the pilot with adherence to 
AFR Rule #4 (i.e., remain FMS-coupled 
whenever possible) by providing a ‘strategic 
reconnect’ capability. This function was to be 
used following an MCP tactical maneuver that 
took the aircraft off its FMS path or cruise 
altitude. AOP would construct a nominal 
reconnect path and probe it for conflicts, 
providing as well the capability to resolve them. 
AOP would also provide the pilot with guidance 
regarding the MCP settings required to again 
become ‘fully coupled’ to FMS guidance. 

Finally, since all aircraft in this experiment 
were AFR, and no aircraft entered terminal 
airspace during the simulation runs, no active 
ATC simulation component was required to 
accomplish the experiment objectives. Though 
ATC would normally assign RTAs dynamically, 
the RTAs were predetermined for this 
experiment and sent as scripted data link 
messages at scheduled times during the 
appropriate scenarios. Each scenario contained 
63 aircraft with destination airports in common 
with the 12 subject pilot ASTORs. These 63 
aircraft were modeled by batch ASTORs flown 
by the pilot model, which enabled them to 
exercise AOP’s ‘Resolve RTA’ capability when 
the situation called for it. 

6 Experiment Participants, Design, and 
Procedure 

6.1 Pilot Participants 

Forty-eight commercial transport pilots 
participated in the NASA Langley HITL 
experiment. However, since two of these pilots 
had not flown a transport category aircraft 
within the last year, data collected from these 
pilots are not included in data analyses. The 
remaining 46 pilot participants consisted of 38 
commercial transport pilots employed by U.S. 
air carriers or aircraft manufacturers and eight 
commercial transport pilots employed by 
European air carriers. The participation of 
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European as well as American pilots was 
desired to support a global perspective on ATM 
research. 

All of the pilots were male and ranged in 
age between 38 and 62 years. Twenty-two of the 
participants were captains, and the other 24 
were first officers. On average, the pilots had 
19 years of airline experience and over 10,000 
hours of airline flying experience. At the time 
of the study, 33 of the participants served as 
Boeing 777 pilots, nine served as 747 pilots, one 
served as a 757/767 pilot, one served as a 
737NG pilot, one served as a 737-300 pilot, and 
one served as an A330 pilot. Pilot recruitment 
favored those having recent experience with 
glass cockpit technology since their experience 
facilitated their training and use of the ASTORs. 

6.2 Experiment Design 

6.2.1 Independent Variables 
Common experiment design matrices were used 
to collect data from pilot participants within the 
NASA Langley ATOL and from controller 
participants within the NASA Ames Airspace 
Operations Lab. A 2x2 within-subject design 
(Pig. 4) was used to collect data during four 
moderate duration (“M” series) 30-minute 
experiment scenarios. The 30-minute scenarios 
involved either the presence (Yes) or absence 
(No) of scheduling assignments (i.e., RTAs) 
provided to aircraft operating within an airspace 
having a sustained traffic density level either 1.5 
times (1.5x) or 2 times (2. Ox) greater than 
current day capacity. 8 


Scheduling Assignment 

No Yes 



30-minute 

30-minute 

1.5x 

Scenario 

Scenario 

Traffic 

“Ml” 

“M4” 

Density 

30-minute 

30-minute 

l.Ox 

Scenario 

Scenario 


“M2” 

“M3” 


Fig. 4 Experiment design matrix 
for 30-minute scenarios. 


A simplified assumption was made for current day 
capacity. ‘l.Ox’ was defined as 18 aircraft per 10,000 
nmi 2 , or approximately 164 aircraft in the airspace region. 


A separate 3x1 within-subject design (Pig. 
5) was used to collect data during three short 
duration (“S” series) 15-minute experiment 
scenarios. During the 15-minute scenarios, 
aircraft operated within an airspace having a 
2. Ox traffic density level, and, in two of the 
three 15-minute scenarios, scripted events that 
manipulated the timing of aircraft trajectory 
changes were introduced. The event was a new 
RTA indicating that a delay maneuver was 
required. The timing was either Dispersed, i.e., 
sent to all 75 ASTOR aircraft at various 
(dispersed) times throughout a scenario, or 
Synchronous, i.e., sent to selected aircraft at the 
same time during a scenario. 

Timing of Trajectory Change Event 



No Scripted 
Trajectoiy 
Changes 

Dispersed 

Trajectory 

Changes 

Synchronous 

Trajectoiy 

Changes 

2. Ox 

15-minute 

15 -minute 

15 -minute 

Traffic 

Scenario 

Scenario 

Scenario 

Density 

“SI” 

“S2” 

“S3” 


Fig. 5 Experiment design matrix 
for 15- minute scenarios. 


6.2.2 Dependent Measures 

During each scenario, the following quantitative 
data were recorded: the 4D flight path history of 
every aircraft, all conflicts and LOS events and 
their associated timelines and details, the auto- 
flight mode changes (e.g., switching from PMS 
to MCP guidance), and all control actions and 
button pushes by the pilots. Data analyses will 
address both safety and efficiency metrics, and 
results will be reported in subsequent papers. 
When considering safety, the primary metric is 
LOS, 9 and each event involving a subject pilot 
will undergo detailed causal analysis for 
contributing factors, including possible pilot 
errors, modeling/simulation anomalies, and 
design issues in AOP and pilot procedures. 
Additional safety metrics of interest include 
pilot response time to conflict alerts [17], the 
proportion of tactical to strategic conflict 


9 Loss of separation or ‘operational error’ was defined to 
be a lateral closest point of approach (CPA) within 4.5 
nmi while within 800 ft vertically. A CPA of 4.5 to 5.0 
nmi, while within 800 ft vertically, was considered a 
‘proximity event’ and tracked separately from LOS. 
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resolution maneuvers, and a general assessment 
of pilot procedural errors. When considering 
efficiency, metrics of interest include the extent 
of deviation from the nominal trajectory, the 
effect of fuel bum, the predicted RTA 
compliance at scenario completion, and the time 
proportion spent in tactical flight (i.e., not 
‘FMS-coupled’). 

Subjective response data were collected 
from pilots via paper-and-pencil questionnaires. 
Following each scenario, the pilots were asked 
to rate the level of workload they experienced 
during the scenario they had just completed 
using the Modified Cooper-Harper (MCH) 
Subjective Workload Rating Scale [18] 10 , and 
they were asked to characterize the acceptability 
of the airborne self-separation procedures and 
automation tool (i.e., the AOP) for that scenario. 
At the conclusion of the experiment, the pilots 
were asked to provide feedback regarding the 
ASTORs and flight scenarios, the experiment’s 
training methods and materials, the AFR 
operational concept, the airborne self-separation 
procedures, and the AOP tool. Finally, the 
pilots participated in an interactive verbal group 
debrief session with the research team. 


minute scenarios twice in random order, 
completed the post-experiment questionnaire, 
and participated in the group debrief session. 

7 Initial Results and Discussion 

Future reports will present quantitative safety 
and efficiency metrics, as these analyses were 
not yet complete at the time of this writing. 
Similarly, the analyses of pilot acceptability 
ratings of the AFR procedures and tool will be 
subsequently published. This paper presents 
initial results regarding the pilots’ subjective 
rating of workload. It also summarizes some of 
the primary discussion points shared during the 
post-experiment group debrief sessions. 

7.1 Workload Ratings 

Pilots used the MCH scale to provide a 
workload assessment after each of the 
experiment’s 14 flight scenarios. Descriptive 
statistics associated with these data are shown in 
Table 1. 

Table 1 Descriptive statistics for MCH 
workload ratings 


6.3 Experiment Procedure 

The 48 subject pilots participated as groups of 
12 in four separate three-day experiment 
sessions conducted in March 2010. Each 
experiment session consisted of a series of 
training exercises involving classroom 
instruction and hands-on simulated flight, 14 
data collection scenarios, questionnaires 
following each scenario, a final ‘post- 
experiment’ questionnaire, and a final debrief 
session involving all 12 pilots and members of 
the research team. During Day 1, the pilots 
completed approximately 5-1/2 hours of 
training. During Day 2, pilots completed the 
four 30-minute scenarios twice in random order. 
During Day 3, pilots completed the three 15- 



30-minute 

scenarios 

15 -minute 
scenarios 

Mean ( M) 

2.41 

2.14 

Standard 
Deviation (SD) 

1.80 

1.33 

Minimum (Min) 

1 

1 

Maximum {Max) 

10 

8 

Sample Size ( N) u 

340 

253 


Mean workload ratings associated with the 
10-point MCH scale indicate that pilots found 
the tasks performed during the scenarios to have 
a difficulty level of “fair” or “mild” and that 
they felt an acceptable level of mental effort was 
required to attain adequate performance. 
Specific events will be analyzed and described 
in subsequent papers to elucidate why, at times, 


10 Use of the MCH scale yields an overall workload rating 
ranging from “1” (indicating that the instructed task was 
very easy and/or highly desirable; operator mental effort 
was minimal; and desired performance was easily 
attainable) to “10” (indicating that the instructed task was 
impossible and could not be accomplished reliably). 


11 Sample sizes of 368 and 276, respectively, were 
anticipated since each of the 46 pilots was asked to 
provide a MCH workload rating after completing each 
scenario. Four pilots did not complete the MCH scale as 
requested, however, resulting in an absence of data. 
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pilots provided high ratings of workload. For 
example, the research team will examine 30- 
minute scenarios having ratings of “10” (which 
indicate that the instructed task was impossible 
and could not be accomplished reliably) and 15- 
minute scenarios having ratings of “8” (which 
indicate that maximum operator mental effort 
was required to avoid large or numerous errors). 

It was not the intent of the researchers to 
compare the workload ratings of the 30-minute 
scenarios with those of the 15-minute scenarios. 
Therefore, analyses of the 30-minute and 15- 
minute scenarios’ workload ratings are 
presented separately. 

7.1.1 30-minute scenarios 

Descriptive statistics associated with the four 
30-minute scenarios’ workload ratings are 
shown in Table 2. 


Table 2 Descriptive statistics for the four 
30-minute scenarios’ MCH workload ratings 



Ml 

M2 

M3 

M4 

M 

2.18 

2.33 

2.73 

2.41 

SD 

1.83 

1.65 

2.18 

1.41 

Min 

1 

1 

1 

1 

Max 

9 

10 

10 

8 

N u 

85 

85 

85 

85 


Statistical analysis was performed using 
the Wilcoxon Test, a nonparametric within- 
subject test appropriate for analyzing two 
related samples of ordinal data [19], A series of 
Wilcoxon Tests revealed a statistically 
significant difference between pilots’ workload 
ratings for scenario Ml as compared to M3 (p 13 
= 0.0341). Pilots found that the combination of 
higher traffic density and an RTA constraint 
increased workload, whereas either effect 
separately did not. It should be noted that the 
pilots were not told the traffic density of each 
scenario or that the density was changing 
between scenarios. 

When workload ratings were averaged 
across the levels of Scheduling Assignment, a 


12 A sample size of 92 was anticipated; however, four 
pilots failed to complete the MCH scale as requested, 
resulting in an absence of data. 

13 A p-value < 0.05 indicates a statistically significant 
difference between sample means. 


Wilcoxon Test revealed no statistically 
significant difference in workload between the 
1.5x traffic density level (represented in 
scenarios Ml and M4) 14 and the 2. Ox traffic 
density level (represented in scenarios M2 and 
M3) 15 (p = 0.2666). When workload ratings 
were averaged across the levels of Traffic 
Density, a Wilcoxon Test revealed that the 
pilots provided a statistically significant lower 
mean workload rating for scenarios flown 
without RTAs (Ml and M2) 16 when compared 
with scenarios flown with RTAs (M3 and M4) 17 
(p = 0.0306). This RTA effect may be related to 
the extra effort required to comprehend the RTA 
data link message, load the RTA in the FMS, 
and execute the change. 

7.1.2 15-minute scenarios 
Descriptive statistics associated with the three 
15-minute scenarios’ workload ratings are 
shown in Table 3. 


Table 3 Descriptive statistics for the three 
15-minute scenarios’ MCH workload ratings 



SI 

S2 

S3 

M 

1.83 

2.30 

2.27 

SD 

1.16 

1.27 

1.50 

Min 

1 

1 

1 

Max 

8 

6 

8 

N u 

84 

85 

84 


Wilcoxon Tests revealed that pilots 
provided a statistically significant lower mean 
workload rating associated for scenario SI than 
for either scenario S2 {p = 0.0021) or scenario 
S3 (p = 0.0088). However, no statistically 
significant difference was indicated between the 
ratings provided for scenarios S2 and S3 (p = 
0.9820). Therefore, the inclusion of a second 
RTA event affected pilot workload, but the 
relative timing of this event among the 75 
aircraft did not affect pilot workload. In the 
self-separation concept, the trajectory 
management system is distributed among the 
aircraft, and it was expected that the timing of 
trajectory-change events would not have an 


14 M= 2.29 , SD = 1.63, Min*# \.Max 9,,V= 170 
15 M= 2.53, SD = 1.94, Min = \,Max = 10, N= 170 
16 M= 2.25, SD = 1.74, Min = l,Max= 10, N= 170 
17 M= 2.51, SD= 1.84, Min = 1 ,Max= 10, N= 170 
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impact on performance, whether these events 
are dispersed in time or synchronous. The 
workload analysis results are consistent with 
this expectation and provide evidence of the 
agility of the AFR concept. 

7.2 Pilots’ Group Comments Regarding the 
AFR Concept 

The group-debrief sessions produced lively 
discussion on several common themes regarding 
the AFR concept in general and its 
implementation in this experiment. Overall, a 
generally positive view of the concept was 
shared, along with some cautionary remarks on 
various aspects. In some cases, these cautionary 
remarks reflected situations in which some 
pilots felt they lost situation awareness, e.g., a 
conflict alert given later than usual. These late 
alerts are being reviewed, and preliminary 
analysis indicates that modeling errors may 
have caused late alerts in certain situations. 
Other cautionary remarks were projections by 
the pilots into areas beyond the experiment 
scope but of relevance to the general concept of 
operations. For example, a general consensus 
was expressed on the opportunity for gaming 
(i.e., acting in a manner to gain advantage or put 
others at a disadvantage), and that some type of 
monitoring and enforcement may be needed to 
ensure the AFR rules are dutifully followed by 
all flights. 

It was noted that the concept would be well 
suited for near-term use in many parts of the 
world where robust ATC services are lacking, 
provided that international standard operating 
procedures are used. Application in the 
domestic U.S. airspace was considered feasible 
above 18000 ft, but concerns were expressed 
regarding viability in flight phases already high 
in crew workload such as the descent phase and 
approaching the arrival metering fix. In this 
experiment, the pilots were not tasked with 
performing these arrival procedures. 

The positive value of a modified Mode 
Control Panel design to suit AFR operations 
was raised, given that the current design reflects 
functionality associated with current-day ATC- 
based operations, such as the altitude clearance 
limit set in the MCP altitude window. It was 


also noted that the single-pilot testing of the 
concept in this experiment precluded the 
opportunity for crew cross-checks of conflict 
resolutions and other trajectory changes, which 
would be paramount for safety. 

Overall, the research team found the group- 
debrief comments from the pilots to be highly 
constructive. Their input will be used to refine 
the AFR operational concept, tools, procedures, 
and training for future experiments. 

8 Conclusions 

A human-in-the-loop simulation experiment 
evaluating airborne self-separation was 
designed with two objectives: (1) use 

experiment design features in common with a 
companion study of ground-based automated 
separation assurance to promote comparability, 
and (2) assess agility of self-separation 
operations in managing trajectory-changing 
events in high traffic density, en-route 
operations with arrival time constraints. The 
experiment addressed the comparability 
objective by using common experiment design 
matrices, identical initial traffic scenarios, and 
jointly defined metrics. While these steps do 
not guarantee truly comparable results, the 
exercise highlights many issues concerning the 
comparison of significantly different operational 
concepts in dissimilar simulation platforms with 
different human subjects (i.e., commercial 
transport pilots vs. air traffic controllers). The 
comparability assessment is reported in a joint 
paper dedicated to this topic [20], 

Self-separation, due to its distributed 
nature and the use of automation, was found to 
support highly dynamic 4D trajectory -based 
operations while maintaining a low mean level 
of pilot-reported workload. Agility was 
indicated by a workload analysis showing no 
significant difference in pilot-reported workload 
between simultaneous vs. dispersed trajectory 
change events. 

As with any significant operational change, 
implementation of an AFR concept can be 
expected to include fundamental changes in 
procedures and equipment, including many 
specific examples suggested by the pilots during 
the debrief sessions. New responsibilities will 
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also require a change in the operational mindset 
of both pilots and controllers, as the context of 
today’s human-centered, ground-based air 
traffic management system evolves toward more 
automation and distribution of critical functions 
like traffic separation. 
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